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ABSTRACT 

Background info-mation is provided on the development 
and activities of Project Cooperation, a demonstration project to 
help institutions improve educational effectiveness by employing 
outcomes measures and assessment strategies. After highlighting the 
various activities of the project (e.g., a national survey, research 
on different plans and models at demonstration sites, summer 
workshops, and institutional effectiveness workshops), the report 
outlines the process by which selected institutions can serve as 
project demonstration sites by developing and implementing model 
student assessment practices using Amencar. College Testing (ACT) 
Program instruments. The outline provides information to help 
institutions: decide upon the "value-added" or "predictive" 
assessment effort to be undertaken; identify the factors critical to 
project success and ways to achieve success; and successfully 
implement the plan. The stages in developing the projeci proposal and 
implementation plan are listed, along with the purposes of each 
stage. Next, the following research models are presented: (l) a 
value-added model using ACT's Collegia e Assessment of Academic 
Proficiency (CAAP) instrument as both pre- and post-test to measure 
students 1 acquisition of general education skills and information; 
(2) a value-added model using ACT'S Assessment Student Skills for 
Entry Transfer (ASSET) instrument for skills assessment at entry and 
CAAP as a measure of value added; and (3) two predictive research 
models using either CAAP or the ACT'S College Outcome Measures 
Project instrument to assess transfer students' general education 
knowledge and predict their four-year college performance. 
Participating institutions and project officers are listed. (JMC) 
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A joint effort of community college educators and 
ACT to answer questions about institutional 



effectiveness and outcomes assessment 
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A RESPONSE TO A NEED : After decades of unprecedented expansion and success in 
providing wider access to higher education, community colleges are being 
called upon to provide objective evidence of student growth and success. As 
the assessment movement progresses in response to these demands, educators are 
Increasingly recognizing the value of outcomes assessment, not only as a way 
of demonstrating results but as a means of improving institutional quality and 
student success. As they consider the variety of options open to them, 
administrators basically want to know how to ask the right questions and use 
the results to improve programs and student persistence. 

Project Cooperation grew out of this interest and concern. In developing a 
position statement on value-added instruction, the National Council of 
Instructional Administrators (NCIA) Task Force on Value-Add*' Education asked 
for ACT 1 s assistance. ACT's subsequent Involvement 1s a natural extension of 
its long tradition of working cooperatively with convunity college 
educators. The long standing cooperative relat1onr,n1p between the National 
Council for Student Development (NCSD) and ACT highlighted the need to 
consider an institution's effectiveness from a student development as well as 
academic perspective and resulted in NCS0 ! s becoming a full partner in the 
Project. The project will bring together a number of community college 
Interest groups and associations resulting in mc^e highly targeted and focused 
services for community college educators. Plans for Project Cooperation 
include developing a national survey on institutional effectiveness and a 
monograph based on survey results; conducting a 1988 Summer Institute on 
Institutional Effectiveness at Howard Community College 1n Columbia, Maryland; 
and establishing a number of model programs using outcomes measurement at 
pilot sites across the country. 



TEAMWORK : Project Cooperation Involves a partnership between the National 
Council for Student Development (NCSD), the National Council of Instructional 
Administrators (NCIA), two Councils of the American Association of Community 
and Junior Colleges (AACJC), and the American College Testing Program (ACT). 



GOALS 

t To develop a position statement that recommends specific models for 
evaluating institutional effectiveness 1n regard to both an 
Institution's general education curriculum and student support services. 

t To provide support to AACJC in Informing community college educators 
about Issues Involved in assessing institutional effectiveness. 



• To provide, as part of NCSO's leadership development series, a Summer 
Institute on Institutional Effectiveness Through Outcomes Assessment for 
community college student services and academic administrators. 

• To establish pilot sites to model recommendations generated by Project 
Cooperation, Activities at these sites will include: 

—Developing institutional effectiveness assessment instruments and 
techniques 

—Piloting an instrument that meets the institutional effectiveness 
needs of community colleges 

—Implementing institutional effectiveness evaluation models at selected 
community colleges to develop and demonstrate successful institutional 
effectiveness evaluation methods 



HOW COMMUNITY COLLEGES CAN PARTICIPATE : Project Cooperation will provide many 
services for community college educators, including: 

• A national survey of community colleges that will gather information 
which will help define institutional effectiveness from both an 
instructional and student services perspective. The survey will also 
help identify successful practices and establish an understanding of 
both current and desired outcomes assessment practices at two-year 
colleges. Results of the survey will be published in cooperation with 
AACJC and cooperating councils. 

• A 1988 Summer Institute focusing on institutional effectiveness through 
outcomes assessment both in the instructional and student services 
areas. 

• A monograph on institutional effectiveness through outcomes assessment 
which will include specific recommendations for the AACJC Board of 
Directors 1 consideration. 

• A series of national seminars on institutional effectiveness through 
outcomes assessment for community college educators sporsored by the ACT 
National Center for the Advancement of Educational Practices during the 
spring of 1989. 



A FOUNDATION FOR THE FUTURE : Project Cooperation brings together NCSD's 
interest in advancing leadership training for student services leaders, NCIA's 
interest in advancing the understanding of community college administrators of 
the issues surrounding assessment, and ACT's interest in supporting the 
development of programs that address the needs of community college 
educators. Through the strengths of its member organizations, Project 
Cooperation will provide a foundation of professional experience, shared 
purposes, and resources that community college educators can draw on in 
developing the most appropriate assessment strategies for their institution. 



FOR MORE INFORMATION : Contact any of the following steering committee 
members: 



NCSD 

Stephen Maier, Co-chair 

Dean of Instruction for Student 

and Community Development and 

President, NCSD 
New Mexico Junior College 
Lovington Highway 
Kobbs, NM 88240 

Walter Bumphus 
Dean of Students and 

President-Elect, NCSD 
Howard Community College 
Little Patuxent Parkway 
Columbia, MD 21044-3197 



NCIA 



Carol J. Viola 

Provost, Open Campus and 

Resident, NCIA 
College of DuPage 
Lambert Road and 22nd Street 
Glen Ellyn, IL 60137 

William Christopher 

Dean for Instruction, 
NCIA Board Liaison, and 
Director of National Issues 

Whatcom Community College 

5217 Northwest Road 

Bellingham, WA 98226 



Robert Keys 

President and AACJC Staff Liaison 
John Wood Community College 
150 S. 48th Street 
Quincy, IL 62301-1498 



Wayne E. Giles 

Vice Chancellor, Educational 

Services and Chair, 

NCIA Task Force on 

Value-Added Education 
Metropolitan Community Colleges 
3200 Broadway 
Kansas City, MO 64111 



ACT 



Richard D. Rowray, Project Administrator 
Executive Director 
ACT National Center 
American College Testing Program 
2201 N. Dodge Street 

PO Box 168 
Iowa City, IA 522^3 




Project Cooperation: A Joint Effort of Community College 
Educators and ACT to Answer Questions About 
Institutional Effectiveness and Outcomes Assessment 



Project Cooperation involves an ongoing partnership initiated in 1988 between 
the National Council for Student Development (NCSD), the National Council of 
Instructional Administrators (NCIA), and the American College Testing Program 
(ACT). It is one of the first, if not the first, major collaborative efforts 
by two councils of AACJC. ACT has worked with AACJC and various others of its 
councils over a number of years. Indeed, it was knowledge of such a working 
relationship between ACT and NCSD that led the head of a special Task Force 
of NCIA to contact ACT for help in 1987. AACJC had charged the NCIA with the 
task of studying the concept of M val ue-added M education. In the process of 
probing this matter of ''value-added, " more questions arose. Ideas started to 
develop, and many ideas for cooperative efforts across several groups began 
to take shape while even more questions arose. As a result. Project 
Cooperation was established. Project Cooperation is a multifaceted approach 
to answering some of those questions by helping to determine the right 
questions to ask and the best uses to make of the answers in order to increase 
institutional effectiveness by assuring student success. 

The entire higher education community is being called upon to provide 
convincing evidence of student growth and success. As the assessment movement 
progresses in response to this challenge, educators are increasingly 
recognizing the value of outcomes assessment as a source of that evidence and 
as a means of improving institutional quality and student success. Indeed, 
this was a part of AACJC's 1988 Public Policy Agenda: to "develop appropriate 
student outcomes measures and promote student success." 

Project Cooperation focuses on the use of student outcome measures for 
assessing institutional effectiveness. The four principal facets of Project 
Cooperation represent four different contexts from which information wiil be 
drawn to address the issue. 

A National Survey 

A survey questionnaire was sent to the CE0 ! s of all members of AACJC in the 
fall of 1988. The objective was to obtain responses to a comprehensive set 
of items that will help determine ,ie student outcomes that are currently 
being measured, which are being used in assessing institutional effectiveness, 
what impacts these uses are having on the institutions, what practices and 
needs seem likely for the near future, and how our institutions have organized 
their assessment efforts. Preliminary results of the survey, which had a 53 
percent response rate, were presented at conferences throughout the nation 
including the AACJC and AAHE meetings. Final results, which will be published 
in November of 1989, will provide a picture of the current status and future 
status of assessment in the two-year college community. 

Demonstration Sites 

The second aspect of Project Cooperation features institutions selected to 
serve as demonstration sites . NCSD and NCIA each nominated institutions, and 
a number of campuses were eventually selected to serve as demonstration sites 
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(see below*). Different plans or models have been developed for using student 
outcomes measures to assess institutional effectiveness and these will be 
tested at the demonstration sites. Basically there are two different types 
of models: a value-added model and a predictive or transfer model which will 
assess students who are ready to transfer to a four-year institution. The 
demonstration site facet of Project Cooperation allows for investigation in 
a real life" situation, and it provides the opportunity for fine tuning our 
instruments and models. 

Project Cooperation Suner Workshops 

The third facet of Project Cooperation involves a workshop setting to which 
individuals who have exhibited a special interest or involvement in this issue 
are invited by NCSD and NCIA. This aspect is an extension of NCSD's week-long 
Leadership Conferences which have had ongoing cooperation from ACT since their 
initiation in 1984, The first of these workshops dedicated to exploring 
issues related to Project Cooperation was hosted in 1988 by Howard Community 
College in Columbia, Maryland. In keeping with the cooperative spirit of 
Project Cooperation, NCIA took the lead in organizing this summer's workshop 
which was held in Nashville July 9-11. Representatives from demonstration 
sites, from the National Council for Research and Planning, and the partners 
involved in Project Cooperation heard from nationally recognized leaders in 
institutional effectiveness, value-added assessment, and student success. 
Demonstration site teams worked toward development of the research design to 
be implemented on their campuses. The 199C workshop is planned for St. Louis. 

Regional Institutional Effectiveness Conferences 

The fourth component provides a medium by which the issues identified and the 
alternatives delineated through other phases of Project Cooperation can be 
shared with colleagues in two-year institutions throughout the nation. The 
ACT National Center devoted its 1989 National Conference Series for Two-Year 
Institutions to Project Cooperation. The conferences were sponsored by ACT 
with participation and collaboration by representatives of NCSD and NCIA. The 
1990 conference series is scheduled for the following cities and dates: 
Baltimore, MD, February 15; Charlotte, N.C., February 27; Irvine, CA, May 1; 
and Detroit, MI, May 17. In the coming year we also anticipate that Project 
Cooperation will be represented once again on national programs such as AACJC 
and the AAHE Assessment Forum. 

Written reports on each of the first three components of Project Cooperation 
will be distributed to AACJC members, and a final report on Project 
Cooperation will also be prepared and distributed. 

Susan Cooper Cowart 
Research Specialist, 
Educational Services 

* Institutions serving as Project Cooperation Demonstration Sites: 
Bakersfield College; Chemeketa Community College; Dyersburg State 
Community College; Howard Community College; Macomb Community College; 
Massachusetts Bay Community College; Metropolitan Community Colleges of 
Kansas City - Longview, Maplewoods, and Penn Valley; Midlands Technical 
College; Orangeburg Calhoun Technical College; Scottsdale Community 
College; St. Louis Community College - Florissant Valley, Forest Park, 
Meramec; Technical College of the LowCountry. 
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AN INTRODUCTION TO THE PROJECT 



The primary purpose of the demonstration site phase of Project Cooperation is to have 
selected institutions develop and implement student assessment practices to a point where 
they could serve as templates for other two-year colleges. This can best be accomplished via 
an energetic collaboration between each demonstration site and ACT. This collaboration 
requires that: 

o each demonstration site will 

a) develop and implement appropriate student outcomes assessment 
practices and procedures, and 

b) pilot student assessment instruments that meet the institutional 
effectiveness needs of community colleges 

o ACT will 

a) provide academic proficiency instruments at a reduced cost, and 

b) provide consultative support 

The collaboration between the demonstration sites and ACT is needed because there is a 
desire in the two-year college sector to generate new student assessment practices and 
implement those practices into the fabric of the institution. Two-year colleges have^ 
expressed a desire for more information on and illustrations of successful assessment 
practices because of the pressures to demonstrate how institutional effectiveness contributes 
to student growth. Following the adoption and implementation of the new practices brought 
forth by the demonstration sites, it is believed that two-year institutions will provide new and 
improved assessment behaviors, processes, and functions. 

The Demonstration Site Outline follows (Section 3) and is offered as a beginning point in the 
dialogue between ACT and those colleges that have been identified as demonstration sites. 

A well-written plan, mutually agreed upon, will help clarify the commitment being made by 
each site and ACT. The plan outlined in Section 3 represents two ingredients required for 
the success of this aspect of Project Cooperation: a desire to innovate through involvement 
and a sustained commitment through planning. 

The plan is driven by the purpose of Project Cooperation~to recommend successful strategies 
and models of student assessment needed for institutional effectiveness. It is offered with a 
vision of the monograph to be written, the jpecific recommendations to be forwarded to the 
AACJC Board ot Directors, and the inquiries coming from colleagues looking for assistance and 
direcnon. This project represents the intent of the demonstration sites and ACT to design 
particular approaches and analyze their contribution to institutional effectiveness. 
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OUTLINE OF THE PROCESS 



In order for the efforts of each demonstration site to be effecdve it is necessary to prepare a 
proposal specific to each site. This outline will serve as a guide to 1) identifying, clarifying, 
and defining the model effort to be undertaken; 2) describing the factors critical to the 
success of the project; 3) outlining the tasks required to fulfill the critical success factors, 
and 4) successfully implementing the plan by monitoring, evaluating, and preparing 
recommendations. To accomplish these ~esults the proposed deveiopment and plan 
implementation is divided into six stages. Stages I, n, and III are designed to help formulate 
a proposal. Stages IV, V, and VI represent the project plan, reporting, and follow-up. 

Project Proposal 

Stage I Information 

o nature of the demonstration site opportunity 

o desire for involvement 

o the five demonstration site models 

o identify a possible model/option 

o review agreement stages 

o establish steps to reach Stage II 

Purposes: 1) To establish open communications with each recommended 
demonstration site and ACT; 2) To identify and select ACT and campus 
project coordinators for each site; 3) To understand the assessment 
opportunity to be pursued and outlined in Stage n. 

Stage II Defining the Project 

o model option selected 
o project title 
o project purpose 
o project outcomes 



Purposes: 1) The selection and clarification of the project to be pursued; 
2) Following tentative approval of the project proposal by the 
demonstration site and ACT person .el, complete the planning stage (Stage 
III) in order to determine if the project is operationally feasible; 3) 
Resolve conflicts over envisioned outcomes, responsibilities, resources, and 
the time schedule. 

Stage III Designing the Project Proposal 
o situational analysis 
o critical success factors 
o preliminary implementation task matrix 

Purposes: 1) Stage III is to further detail the proposal identified in Stage 
II and to determine if the project is go or no-go; 2) Conduct a "gap 
analysis" between what is and what should be and determine if Stage III 
fills the gap; 3) Turn Stage III into an explicit Action Plan that is 
supported unequivocally by all key parties. (Note: In this stage the 
proposal converts to an agreed upon plan.) 
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Pro ject Plan 

Stage IV Implementation and Evaluation Matrix 
o monitoring and control 
o project modifications 
o project progress report 
o project evaluation 

Purposes: 1) Jointly monitor and control the flow of the project; 
2) Identify and document project modifications and adjustments; 3) Check 
adherence to time and budget schedules; 4) Evaluate the model undertaken and 
document results and recommendations; 5) Avoid surprises and crises; rN Ensure 
project implementation that produces "models" by describing exemplar) use of ACT 
programs and services; and 7) Document on-campus processes required to establish 
and implement successful assessment practices. 

Stage V Written Report(s) and Releases 

o Preparation of written report(s) including: 

(a) documentation of the process (required and optional topics) 

(b) results of the research 

(c) recommendations (based on Stages I-IV) 
o Distribution of final report and product 

Purposes: 1) Address the major process variables and document the results; 
2) Develop documentation related to key issues critical to assessing 
institutional effectiveness and the use of the information; 3) Prepare and 
release to the AACJC Board of Directors a monograph reflecting the 
results and recommendations; 4) Use project findings in the summer 
conference, the national seminar series and other settings; and 5) 
Determine next steps with each site. 

Stage VI Follow-up 

o project history 
o project emulation 

Purposes: 1) To determine, after a period of 2-3 years, what new practices 
h?.ve become embedded in the institution; 2) Determine whether other 
institutions will profit from this demonstration site experience. 
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RESEARCH MODEL FOR TWO-YEAR INSTITUTION WITH 
SUPPLEMENTAL OPTIONS 



Each institution involved with the demonstration site project agrees to participate in a basic 
research model consisting of the use of at least one or two ACT programs related either to a 
"value-added" or a "predictive" study. In addition, institutions will be given the opportunity 
to extend the basic design to at least one additional related topic of mutual concern in such 
areas as assessing student development outcomes, studying the utility of CAAP as a transfer 
support tool, carrying out an image analysis, developing a secondary school outreach program, 
or developing an adult, business/industry outreach program. Institutions will be given a 
number of choices in the demonstration site project, but ACT will have a coordinating role in 
order to ensure that all models are represented- It is also intended that the model sites will 
address the research projects from one or more perspectives such as: age, sex, ethnicity, 
special populations, and program type. Thus the demonstration sues should be considered 
models that will be of maximum benefit to all community colleges. 
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DEMONSTRATION SITE CAAP-CAAP RESEARCH MODEL 

Purpose: Two-year institutions are committed to contributing to student learning and 
development, and they need to be able to show that they are achieving this mission. 
This research is aimed at demonstrating a procedure through which convincing 
evidence of institutional effectiveness can be provided In this case the evidence of 
institutional effectiveness focuses on change in student learning and development 
that may take place between time of entry and time of "exit," i.e., program 
completion at a two-year institution or transfer to a four-veai institution. 
Research of this type may be labeled "student growth" or "•'alue-added." 

A "value-added" research model requires that a base of information be collected for 
students at the time they enter the two-year institution in order to establish a 
measure against which to compare information collected at the time students are 
ready to leave the institution. This base may serve a variety of other purposes as 
well: determination of whether remedial instruction is needed, placement of 
students m regular collegiate-level courses, evaluation of student needs, and 
evaluation of the level of student development at time of entry. 

ACT offers numerous instruments to help in establishing this base of information, 
and each demonstration site will be using some of these assessment instruments for 
this purpose. 

The Model: The CAAP-CAAP research model uses the Collegiate Assessment of Academic 
Proficiency-ACT's instrument designed specifically to assess outcomes. This model 
provides the most direct measure of "value-added" with respect to student 
acquisition of information and skills included in a general education curriculum. 
CAAP is modular in form and includes a module for each of the core areas of a 
college curriculum: reading, writing, mathematics, critical linking, and science 
reasoning. CAAP is designed to assess attainment of general foundational skills 
typically attained within the first two years of college. The research design will 
determine the number and choice of modules to be administered. 

A student's score at or near time of exit is expected to differ from that attained 
on an entry assessment It is reasonable to expect that difference to reflect 
growth in a student's knowledge and .kills related to a general education 
curriculum. That is to say that the exit-time score may be viewed as being a 
function of the student's score a' entry, the curriculum taken at the two-year 
institution, the student's performance in these courses, and the personal non- 
cognmve development that takes place during the period between entry and program 
completion. ' * 5 

Performance on the entry assessment is most likely a function of high school 
courses taken and psrformar.ee in those courses for students entering shortly after 
high school completion. This is likely to be a less significant impact on entry 
assessment scores of students who are entering after a few years of work, military 
service, or other similar activities. 
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The difference between entry scores and exit scores c?ji be seen to be a function 
of all these factors: high school performance, activity since high school, courses 
completed at the two-year institution and performance in those courses, and some 
development factor that relates to personal growth and maturity that takes place 
during the collegiate experience. 

The Sample: In order to fulfill the requirements of this research, it is essential that the 
sample be selected according to specific, predetermined criteria. ACT asks that the 
sample be from among the student population enrolled in a degree program, transfer 
program, or a shorter program of study that is less than two years but more than a 
single semester. Further, the sample is to be of students who are enrolling in a 
program of study that has SQXDC general education component as a requirement 
This requirement simply assures that students included in the sample have taken at 
least some of the courses that the assessment instrument targets. In addition, 
each institution will be asked to sample for a focused investigation of either 
gender, age, race/ethnicity, or educational objective, ACT will assist the institution 
in determining how to draw the sample. 

A high rate of sample attrition may be anticipated during the period between entry 
testing and "exit" testing. In order to assure an adequate sample size for students 
who are near program completion, institutions should plan to test on a ratio of 4 to 
1, i,e, four entering students for each "exiting" student needed This ratio may be 
adjusted if institutional retention/attrition data would argue for either an increase 
or decrease, A sample size of either a minimum of 100 students or approximately 
10% of the group of students at or near program completion-whichever is larger-is 
desired for the "exit" test sample. Sites involved in research on a target student 
population are advised that these same criteria should hold for the focused 
subsample: a ratio of four entering students for each exiting student to be in the 
subsample with a minimum being the larger of either 100 students or 10% of the 
target populatior. The model uses a longitudinal design in that the exit sample is 
from the sample of students tested upon entry. 

Additional Data Needs: Basic demographic information on students will be collected on 
CAAP instrument These items include sex, age, race/ethnicity, program/major at 
the two-year institution, whether English is the student's native/first language, and 
previous post-secondary enrollment. 

High school transcript information is needed as well as complete transcript 
information from the two-year institution on courses completed, course grades, and 
hours per semester through time of exit 

A survey will be administered to students to provide one indication of student 
development and satisfaction with institutional attributes. An ACT survey will be 
available to obtain information needed for this aspect of u. j research. 



DEMONSTRATION SITE ASSET-CAAP RESEARCH MODEL 



Purpose: Two-year institutions arc committed to contributing to student learning and 
development, and they need to be able to show that they are achieving this mission. This 
research is aimed at demonstrating a procedure through which convincing evidence of 
institutional effectiveness can be provided In this case the evidence of institutional 
effectiveness fty^r s on change in student learning and development that may take place 
between time 01 . and time of "exit," i.e. f program completion at a two-year institution or 
transfer to a four-year institution. Research of this type may be labeled "student growth" or 
"value-added." 

A "value-added" research model requires that a base of information be collected for 
students at the time they enter the two-year institution in order to establish a 
measure against which to compare information collected at the time students are 
ready to leave the institution. This base may serve a variety of other purposes as 
well: determination of whether remedial instruction is needed, placement of 
students in regular collegiate-level courses, evaluation of student needs, and 
evaluation of the 'evel of student development at time of entry. 

ACT offers numerous instruments to help in establishing this base of information, 
and each demonstration site will be using some of these assessment instruments for 
this purpose. 

The Model: The ASSET-CAAP reseaich model uses ASSET and the Collegiate Assessment of 
Academic Proficiency (CAAP)--ACTs instrument designed specifically to assess outcomes. 
ASSET is ACT's entry skills assessment instrument designed for course placement in 
two-year institutions. Almost one-third of the two-year student population is tested 
annually with ASSET, This model provides the opportunity to determine whether this 
commonly used entry assessment instrument (ASSET) might be coupled with CAAP to 
measure "value-added" with respect to student acquisition of information and skills 
included in a general education curriculum. If this proves to be feasible, this finding 
will mean that institutions that use ASSET for placement purposes will need no additional 
entry assessment in order to obtain a measure of value-added for students who have 
completed their program of study. 

For purposes of this research, both ASSET and CAAP must be administered as entry 
assessment instruments, with only CAAP to be administered at or near time of exit. 
CAAP is modular in form and includes a module for each ot the core areas of a 
college curriculum: reading, writing, mathematics, critical thinking, and science 
reasoning. CAAP is designed to assess attainment of general foundational skills 
typically attained within the first two years of college. The research design will 
determine the number and choice of modules to be administered in conjunction with 
the ASSET. 
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A student's score at or near time of exit is expected to differ from that attained 
on an entry assessment It is reasonable to expect that difference to reflect 
growth in a student's knowledge and skills related to a general education 
curriculum. That is to say that the exit-time score may be viewed as being a 
function of the student's score at entry, the curriculum taken at the two-year 
institution, ihe student's performance in these courses, and the personal non- 
cognitive development that takes place during the period between entry and program 
completion. 

Performance on the entry assessment is most likely a function of high school 
courses taken and performance in those courses for students entering shortly after 
high school completion. Thi3 is likely to be a less significant impact on entry 
assessment scores of students who are entering after a few years of work, military 
service, or other similar activities. 

The difference between entry scores and exit scores can be seen to be a function of all 
these factors: high school performance, activity since high school, courses completed at 
the two-year institution and performance in those courses, and some development factor 
that relates to personal growth and maturity that takes place during the collegiate 
experience. In this research the difference between the CAAP exit score and the CAAP 
entry score estimated from the ASSET entry score will be computed and compared to the 
difference between the actual CAAP entry score and the CAAP exit score. This is 
designed to provide the basis for determining the form of the statistical relationship 
between ASSET and CAAP that will indicate whether ASSET can be substituted for CAAP 
as the entry instrument in a value-added model. 

The Sample: In order to fulfill the requirements of this research, it is essential that the 
sample be selected according to specific, predetermined criteria. ACT asks that the 
sample be from among the student population enrolled in a der ,e program, transfer 
program, or a shorter program of study that is less than two years but more than a 
single semester. Further, the sample is to be of students who are enrolling in a 
program of study that has sflmc general education component as a requirement. 
This requirement simply assures that students included in the sample have taken at 
least some of the courses that the assessment instrument targets. In addition, 
each institution will be asked to sample for a focused investigation of ei;her 
gender, age, race/ethnicity, or educational objective. ACT will assist the institution 
in determining bow to draw the sample. 

A high rate of sample attrition may be anticipated during the period between entry 
testing and "exit" testing. In oider to assure an adequate sample size for students 
who are near program completion, institutions should plan to test on a ratio of 4 to 
I, i.e. four entering students for each "exiting" student needed. This ratio may be 
adjusted if institutional retention/attrition data would argue for either an increase 
or decrease. A sample size of either a minimum of 100 students or approximately 
10% of the group of student at or near program completion-whichever is larger-is 
desired tor the "exit" test sample. Sites involved in research on a target student 
population are advised that these same criteria should hold for the focused 
subsample: a ratio of four entering students for ^ach exiting student to be in the 
subsample with a minimum being the larger o r either 100 students or 10% of the 
target population. The model uses a longitv ..nal design in that the exit sample is 
from the sample of students tested upon entry. 

ERIC 
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Additional Data Needs: Basic demographic information on students will be collected on both 
the ASSET and the CAAP instruments. These items include sex, age, race/ethnicity, 
program/major at the two-year institution, whether English is the student's 
native/first language, and previous post-secondary enrollment 

High school transcript information is needed as well as complete transcript 
information from the two-year institution on courses completed, course grades, and 
hours per semester through time of exit. 

A survey will be administered to students to provide one indication of student 
development and satisfaction with institutional attributes. An ACT survey will be 
available to obtain information needed for this aspect of the research. 
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DEMONSTRATION SITE TRANSFER RESEARCH MODEL USING CAAP 



Purpose: Two-year institutions are committed to contributing to student learning and 
development, and they need to be able to show that they are achieving this mission. 
This research is aimed at demonstrating how a measure of student learning outcomes 
can be used to predict student success in a subsequent learning environment In 
this case the evidence of institutional effectiveness focuses on the performance of 
two-year college students in their junior year at a four-year institution. 

A predictive research model requires that a base of information be collected near 
time of exit for students who intend to transfer to a four-year institution. This 
base is needed in order to establish a measure against which to compare information 
collected on the students after one year of study at the transfer institution. If the 
CAAP scores appear to be reliable predictors of student performance at the four- 
year institution, then these scores might come to assist in the admissions decision 
as well as to facilitate the transition of students to the four-year institution in the 
absence of formal or applicable articulation agreements. 

The exit data would be most useful to two-year institutions in the assessment of 
many attributes of their institution, for example: to evaluate strengths and 
weaknesses of the auricular offerings and content at two-year institutions, to 
delineate essential courses for students to take as a part of the general education 
requirements, and to possibly modify the general education component of various 
degree or program requirements. 

The Model: The CAAP predictive research model uses the Collegiate Assessment of Academic 
Proficiency-ACTs instrument designed specifically to assess outcomes. CAAP is 
modular in form and includes a module for each of the core areas of a college 
curriculum: reading, writing, mathematics, critical thinking, and science reasoning. 
CAAP is designed to assess a'tainment of general foundational skills typically 
attained within the first two years of college. The research design will determine 
the number and choice of modules to be administered. 

A student's score at or near time of exit is expected to be a function of the 
curriculum taken at the two-year institution, the student's performance in these 
courses, and the personal non-cognitive development that takes place during the 
period between entry and program completion, and a set of demographic factors 
such as age, gender, and race/ethnicity. 

Performance at the four-year institution is similarly modeled as being a function of 
scores on the CAAP plus curriculum taken at the four-year institution, performance 
in these courses, personal non-cognitive development that will take place during this 
period, and a set of basic demographic factors such as age, sex, race/ethnicity. In 
addition, some institution-specific attributes may be expected to impact on the 
student's performance. Demonstration sites will collaborate with the ACT consultant 
to develop an operational definition of "success. 11 
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The Samp e: In order to fulfill the requirements of this research, it is essential that the 
sample be selected according to specific, predetermined criteria. ACT asks that the 
sample be from among the student population planning :o transfer to a four-year 
msntuticn and nearing completion of their degree or program of study at the two- 
year institution. Further, the sample is to be of students who are enrolled in a 
program of study that has some general education component as a requirement. 
This requirement simply assures that students included in the sample have taken at 
least some of the courses that the assessment instrument targets. In addition, 
each institution will be asked to sample for a focused investigation of either 
gender, age, race/ethnicity, or educational objective. ACT will assist the institution 
in determining how to draw the sample. 

A high rate of sample attrition may be anticipated during the period between exit 
testing and completion of the junior at the four-year institution. In order to assure 
an adequate sample size for students who are near completion of their junior year 
institutions should plan to test on a ratio of 4 to 1, i.e. four exiting students for 
each junior year student needed. This ratio may be adjusted if institutional 
retention/attrition data would argue for either an increase or decrease. A sample 
size of either a niinimum of 100 students or approximately 10% of the group of 
students meeting these criteria-whichever is larger-is desired for the "exit" test 
sample. Sites involved in research on a target student population are advised that 
these same criteria should hold for the focused subsample: a ratio of four entering 
students for each exiting student to be in the subsample with a minimum being the 
larger of either 100 students or 10% of the target population. 

Additional Data Needs: Basic demographic information on students will be collected on the 
CAAP instrument. These items include sex, age, race/ethnicity, program/major at 
the two-year institution, whether English is the student's native/first language, and 
previous post-secondary enrollment. 

Demonstration site institutions will have to procure junior year transcripts from the 
four-year institution. The demonstration site institution will be responsible for 
obtaining this information, although ACT will provide a statement of purpose and 
research 'resign to assure the four-year institution of confidentiality o>" he student 
records. Further, a waiver statement will be provided for students to sign stating 
that, for purposes of this research, permission is granted to obtain transcript 
information from the institution to which the student subsequendy transfers. More 
than one transfer institution can be included in the research if that is necessary in 
order to secure an adequate participation rate. 

A survey will be atiiriinistered to students to provide one indication of student 
development and satisfaction with institutional attributes. An ACT survey will be 
available to obtain information needed for this aspect of the research. 
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DEMONSTRATION SUE TRANSFER RESEARCH MODEL USING COMP 



Purpose: Two-year institutions are committed to contributing to student learning and 
development, and they need to be able to show that they are achieving this mission. 
This research is aimed at demonstrating how a measure of student learning 
outcomes can be used to predict student success in a subsequent learning 
environment In this case the evidence of institutional effectiveness focuses on 
the performance of two- year college students in their junior year at a four-year 
institution. 

A predictive research model requires that a base of information be collected near 
time of exit for students who intend to transfer to a four-year institution. This 
base is needed in order to establish a measure against which to compare information 
collected on the students after one year of study at the transfer institution. If the 
COMP scores appear to be reliable predictors of student performance at the four- 
year institution, then these scores might come to assist in the admissions decision 
as well as to facilitate the transition of students to the four- year institution in 
the absence of formal or applicable articulation agreements. 
The exit data would be most useful to two-year institutions in the assessment of 
many attributes of their institution, for example: to evaluate strengths and 
weaknesses of the cunicular offerings and content at two-year institutions, to 
delineate essential courses for students to take as a part of the general education 
requirements, and to possibly modify the general education component of various 
degree or program requirements. 

The Model: The COMP predictive research model uses the College Outcome Measures Project 
instrument which was designed specifically to assess outcomes associated with 
acquisition of knowledge and skills generally attained upon completion of general 
education requirements. Demonstration sites will administer COMP at or near time 
of transfer to a four- year institution. 

A student's score at or near time of exit is expected to be a function of the 
curriculum taken at the two-year institution, the student's performance in these 
courses, and the personal non-cognitive development that takes place during the 
period between entry and program completion, and a se' of demographic factors 
such as age, gender, and race/ethnicity. 

Performance at the four- year institution is similarly modeled as being a function of 
scores on the COMP plus curriculum taken at the four-year institution, performance 
in these courses, personal non-cognitive development that will take place during this 
period, and a set of basic demographic factors such as age, sex, race/ethnicity. In 
addition, some institution-specific attributes may be expected to impact on the 
student's performance. Demonstration sites may add up to 50 items to the COMP 
form. These items allow the institution to focus attention on factors that seem to 
affect students in their life experiences. The demonstration site, in collaboration 
with the ACT consultant, will develop an operational definition of "success" against 
which to compare results of the research. 
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The Sample: In order to fulfill the requirements of this research, it is essential that the 
sample be selected according to specific, predetermined criteria. ACT asks that the 
sample be from among the student population planning to transfer to a four-year 
institution and nearing completion of their degree or program of study at the two- 
year institution. Further, the sample is to be of students who are enrolled in a 
program of study that has some general education component as a requirement. 
This requirement simply assures that students included in the sample have taken at 
least some of the courses that the assessment instrument targets. In addition, 
each institution will be asked to sample for a focused investigation of either 
gender, age, race/ethnicity, or educational objective, ACT will assist the institution 
in determining how to draw the sample, 

A high rate of sample attrition may be anticipated during the period between exit 
testing and completion of the junior at the four-year institution. In order to assure 
an adequate sample size for students who are near completion of their junior year, 
institutions should plan to test on a ratio of 4 to 1, i,e, four exiting students for 
each junior year student needed This ratio may be adjusted if institutional 
retention/attrition data would argue for either an increase or decrease, A sample 
size of either a minimum of 100 students or approximately 10% of the group of 
students meeting these criteria- whichever is larger-is desired for the "exit" test 
sample. Sites involved in research on a target student population are advised that 
these same criteria should hold for the focused subsample: a ratio of four entering 
students for each exiting student to be in the subsample with a minimur being the 
larger of either 100 students or 10% of the target population. 

Additional Data Needs: Basic demographic information on students will be collected on the 
COMP instrument 

Demonstration site institutions will have to procure junior year transcripts from the 
four-year institution. The demonstration site institution will be responsible for 
obtaining this information, although ACT will provide a statement of purpose and 
research design to assure the four-year institution of confidentiality of the student 
records. Further, a waiver statement will be provided for students to sign stating 
that, for purposes of this research, permission is granted to obtain transcript 
information from the institution to which the student subsequently transfers. More 
than one transfer institution can be included in the research if that is necessary in 
order to secure an adequate participation rate. 

A survey will be administered to students to provide one indication of student 
development and satisfaction with institutional attributes. An ACT survey will be 
available to obtain information needed for this aspect of the research. 
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Supplemental Research Options for Demonstration Sites 



Each demonstration site will be required to conduct research in one of the following areas. 
The details of the research will be worked out in consultation with ACT and will be mutually 
agreeable. 

1. St udent Development 

Statement of Purpose: To determine growth in student development that may take place 
between entry and program completion. 

Requirements of the Studv: 1, Define the critical student development outcomes to 
be analyzed Suggestions include the following: (a) determining whether students 
completing remedial/development studies are prepared for regular college-level 
course work (b) the impact of some more intensive/extensive academic advising 
program for students in the research (c) "he impact of a faculty (or staff) 
mentoring program on students in the research (d) the impact of a study skills 
course (e) the impact of an orientation course (f) a retention project 

2. Identify and develop the means of assessing these outcomes. Some 
ACT programs and services that may be useful include: 

a. Evaluation/Survey Services Instruments 

b. Study Skills 

c. DISCOVER 

d. Career Planning Program 

e. ASSET 

3. Implement and evaluate the assessment 

4. Document the process 

2. Articulation 

Statement of Purpose: Two research options are related to the ''articulation" 
research project: one involves articulation between two-year and four-year 
institutions and the other involves articulation between high schools and two-year 
institutions. The former will be referred to as "transfer" and the latter as 
"secondary school outreach." 

Transfer To determine the utility of CAAP as a transfer support tool 
and the impact of the community college transfer program of study on 
the subsequent success of a student at the four-year institution. 

Secondary School Outreach: To impact the college-ready pool and/or 
institutional enrollment as a form of student development outcome or 
community impact outcome and assess results. 



Requiremen ts of the Study: 

Transfer: Define the outcomes that are critical to academic success 
after transfer, assess these outcomes using the CAAP and other 
instruments or documentation; evaluate these approaches; and document 
the process. 
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(a) Administer all CAAP modules to a representative sample 

ofat least 100 students at or near completion of 
their program of study or a 10% sample from this 
group, whichever is larger. 

(b) Provide a transcript for coursework completed 
through the end of the program at the demonstration 
site. 

(c) Procure a transcript for coursework completed 
through the end of the junior year at the transfer 
institution. 

Please see the Student Development option for a listing of ACT programs 
and services that might be useful in this research as well. 

Secondary School Outreach; Define the outcomes that are critical to 
institutional enrollment; assess these outcomes using the P-ACT + and 
other instruments or documentation; evaluate these approaches: and 
document the process. 

(a) Administer the P-ACT* to a mutually approved 
sample of students in key feeder high schools. 

(b) Administer the High School S tudent Needs 
Assessment Survey to students in the tenth grade. 

(c) Administer the Survey of Postsecondary Plans to 
this same population in their senior of high school. 

(d) One year later, survey these students with the high 
School Follow-Up Survey 

(e) Procure a complete, end of senior year high school 
transcript 

(f) Provide a transcript of the first post-secondary 
coursework for each student. 

Please see the Student Development option for a listing of ACT programs 
and services thai may be usefui to this research. In addition, the 
Enrollment Information Service is of particular relevance to this research 
option. 

Adult. Business/Industry Outreach 

Statement of Purp^fir To impact adults, business/industry personnel enrollment as a 
torm ot student development outcome or community impact outcome and assess the 
results at either (a) the end of program completion or (b)completion of a short term 
ot study. 

Requirements Qf the Study; Define the outcomes that are critical to adult or 

a a L n rr UStry P ersonnel enrollment; assess these outcomes using ACT surveys 
tne LAAP (for students in longer-term programs), and other instruments or 
documentation; evaluate these approaches; and document the process. 
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Requiremen ts (continued^ 



(a) Administer the ACT Adult Learner Needs Assessment Survey to 
key components identified for the research. 

(b) Administer placement instruments as required, e.g. ASSET. 

(c) Administer other surveys as required. 

(d) Assess career needs using DISCOVER, CPP, or other 
instruments, as required. 

(e) One year after program/study completion, survey businesses/industry 
to obtain feedback on effectiveness of collegiate program with 
respect to employee performance. 

(0 One year after program/study completion, survey all those who 
completed the program/short-term of study with the Alumni 
Survey for Two- Year In' ntutions. 

Image Analysis 

Statement of Purpose: To develop a means of documenting institutional 
effectiveness and communicating this to targeted audiences. 

Requirements of the Study: Define the factors that are critical to documenting the 
institutional image and communicating this image; assess tnese factors; evaluate the 
process; and document the process. 

Possible target audiences for an image analysis include: 

(a) the secondary education community 

(b) the post-secondary education community 

(c) the local/regional community in general 

(d) business/industry 

(e) government/the public sector 
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Requirements: Each demonstration site must produce written documentation of the 
project This documentation is to include the major process variables involved. At 
the outset these areas are to be defined clearly and the methods of documentation 
and evaluation specified. These process variables are: 

1. Faculty/staff involvement and commitment including how it is 
obtained and how it is manifested operationally. 

2. Procuring and sustaining student involvement and valid 
participation. 

3. Specification of "effectiveness criteria" and operational 
definitions of these criteria, including data generated by ACT 
instrumentation as well as other data* 

4. Using "effectiveness" information for both institutional decision-making 
and external report. 

Options: Each site will be requested to develop materials (either independently or 
in cooperation with other institutions) on one or more issues critical to assessing 
institutional effectiveness and the use of the information. Each site will be 
expected to determine which issue(s) it will focus upon prior to beginning the 
implementation of the project plan and communicate this to ACT. ACT, in turn, 
will play a quality assurance role related to these options. Upon completion of - 
these materials, ACT will facilitate in their publication. Each demonstration site 
may choose from among the following projects: 

1 . Development of case study documentation. 

2. Development of a faculty handbook for guiding involvement in the 
assessment of institutional effectiveness. 

3. Development of a handbook linking assessment of institutional 
effectiveness to institutional decision-making in relation to academic 
programming. 

4. Development of a handbook linking assessment of institutional 
effectiveness to institutional decision-making in the affective sphere of 
student development. 

5. Development of a resource handbook for securing and maintaining student 
involvement in assessing institutional effectiveness. 

6. Development xj! a resource book to document effective methods of 
communicating the results of institutional effectiveness studies to the relevant 
external populations (e.g., legislatures, governing boards, accrediting agencies, 
media, etc.). 

7. Development of a resource handbook for administrative and 
governing bodies for managing and utilizing the assessment of the 
institutional effectiveness process. 
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